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Abstract 

The fusion of radar and electro-optic (E-O) sensor images presents unique challenges. The two 
sensors measure different properties of the real three-dimensional (3-D) world. Forming the 
sensor outputs into a common format does not mask these differences. In this paper, the 
conditions under which fusion of the two sensor signals is possible are explored. The program 
currently planned to investigate this problem is briefly discussed. 

Introduction 

Westinghouse has been developing novel adverse weather landing aids for commercial and 
military aircraft. We have concluded that it will be necessary to use a multiple sensor suite to 
provide both an active radar imaging sensor, and a passive imaging E-O sensor. The radar 
imager provides excellent penetration of adverse weather, but has limited angular resolution. 
The E-O sensor provides very good angular resolution but is severely affected by adverse 
weather such as fog, rain or snow. The fundamental property that distinguishes the two sensor 
classes is operating wavelength. This is both the driver on adverse weather penetration, and 
the driver on angular resolution. When the wavelength is greater than the size of atmospheric 
aerosols and raindrops, the penetration is good. When the wavelength is small compared to the 
receiving aperture, the resolution is good. 

For the current paper, an equally important distinction is the difference between active sensing 
and passive sensing. An active sensor provides its own illumination of the scene to be imaged, 
while a passive sensor depends on either some external illuminator, or on self-emitted radiation 
of the objects being imaged. An active sensor has an advantage in that the properties of the 
illuminating waveform can be exploited for coherent detection of reflected energy. This 
dependence on reflected (i.e. back scattered) energy determines how the active sensor images 
a real 3-D scene. Specifically, electromagnetic properties that are determined by the surface 
to some depth are important in determining the reflection characteristics. In addition, 
macroscopic scale features are important since energy can experience multiple reflections before 
being returned to the receiver. 

For the E-O sensor the considerations are very different. Few surfaces are optically smooth. 
Thus the behavior of such surfaces in reflected light is significantly different than the behavior 
of the self-emitted energy. Multiple reflections of emitted or reflected energy play a minimal 
role in determining signal. The properties that determine reflection or absorption are not well 
correlated with the bulk properties that determine reflection and absorption at radar wavelengths. 


225 



The third distinction between radar and E-0 sensors is that different evolutionary paths have 
resulted in radar providing very precise range and range rate measurements with only limited 
emphasis on received signal strength which is the only property usually quantified with an E-0 
sensor. For the application at hand, the radar image is returned as a range versus azimuth 
angle using an antenna that is mechanically scanned, and which has a shaped beam pattern 
designed to minimize the variation of signal with elevation angle, under the assumption of flight 
nearly parallel to the ground where returns originate. The use of range versus angle as opposed 
to signal return level versus angle presents some challenges. Height of a return source above 
terrain is lost. Converting from an azimuth/range/intensity image to an 
azimuth/elevation/intensity image requires an assumption about the height of the return sources. 
Figure 1 shows an E-0 sensor image of a runway at the Salsbury MD airport. Figure 2 shows 
the same runway as viewed using an X-Band (10 GHz) radar operated in the Monopulse Ground 
Map (MGM) mode. Figure 2 was derived from Figure 3 (azimuth/range/intensity) by assuming 
that all reflecting elements are in a ground plane which has a known orientation with respect to 
the flight path. As shown in Figure 4, each range cell in the radar return is assigned an 
elevation angle on the basis of the aircraft height above the ground plane. While there are a 
number of important error sources which must be accounted for in this process, for the purpose 
of this paper, it is sufficient to assume that those difficulties will be overcome, and that a proper 
image in angle/angle/intensity format will be achieved. 

Fusion Technique 

Westinghouse has approached the task of Radar E-O image fusion as an evolution of previously 
developed technology. The MGM mode for the radar, coupled with a transformation from 
azimuth/range to azimuth/elevation produces an image which has a compatible format with 
standard E-0 images and displays. Westinghouse has also been participating with the David 
Samoff Research Center in a program that uses pyramid decomposition of visible and IR E-0 
images to construct fused images. That program has advanced to the point where real time 
operation at television rates and resolutions will be possible in the very near future. Combining 
these two developments provides a path to the desired Radar E-O fusion. The paper by Dr. 
Hannah at this workshop describes the pyramid fusion technique for visible and IR images. The 
interested reader will find additional information in references 1-3. 

Figure 5 shows the general arrangement of a postulated Radar E-0 image fusion system. The 
Radar is operated in the MGM mode and creates angle/range/intensity images at a low frame 
rate. These are converted to angle/angle/intensity images using a combination of on-board 
inertial and altitude sensors. The images are used to generate a 30 Hz image stream by motion 
compensation plus image extrapolation. This step may occur either before or after pyramid 
decomposition, depending on engineering details. The Radar images are decomposed using 
pyramid decomposition. The E-0 images are similarly decomposed, so that features from both 
images can be identified, matched, and registered. Feature blending/ selection is used to 
produce the composite image in transform space. This image is then inverse transformed, using 
the merged pyramids to construct the angle/angle/intensity image. Standard processes, such as 
gain and level adjustment, are then used to correct that image prior to display. 
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The pyramid decomposition has the effect of generating intermediate images which contain a 
limited range of spatial frequencies. Thus, the decomposition of a high resolution E-0 image 
will result in transformed images that have resolution compatible with the Radar resolution. By 
suitable choice of scan angles, sampling rates, and optical design, the reduced resolution image 
will match the resolution of the Radar such that direct comparison and fusion of features will 
be possible. Figure 6 shows the decomposition and feature match processes. The fusion 
process, represented by a single block, is a variant of the previously published work. 

Each cycle of the pyramid decomposition produces a bandpass image (the Laplacian) that 
contains one octave of spatial frequency data, plus a residue image that contains all spatial 
frequencies from zero to the lower limit of the bandpass image. The two image sources can, 
by suitable choice of sampling grids, provide bandpass images that share a common range of 
spatial frequencies. It is also a property of the pyramid decomposition process that the spatial 
coordinates of each feature are preserved in the transform process. Thus, each feature will be 
represented by both spatial coordinates and spatial frequency content. Relatively simple 
operations such as rectification and threshholding permit the determination that the feature is 
present. If such a test is satisfied in both images, then the features can be fused into a single 
feature that can be displayed. In addition, a feature present in one image, but not the other, 
can be used in the composite image. This will provide an image containing the information 
from both sources. 

Fusion Issues 

The above discussion of Radar E-0 fusion has glossed over several potential difficulties. The 
most obvious is commensurability. Are the features in a Radar image sufficiently similar in 
size, shape, location, or intensity to be clearly identifiable as the same feature by some analytic 
rule? Is the only answer to this question anecdotal, or is there a formal method for resolving 
this issue? 

One approach to the commensurability is shown in Figure 7. Both scenes are derived from the 
same 3-D real world. Each of the sensors has performed a transform into one or more spaces 
depending on where we choose to view the image. If we can add a transform to one or both 
images which produces intermediate images which are demonstrably the same for equal real 
world inputs, then, in that transform space, they are commensurable and can be merged. As 
inspection of Figure 7 shows, it is a generalization of Figure 5 which is the particular transform 
path we are exploring. 

Another issue might be called "fusability". If we identify a feature from both sensors, and can 
conclude that it is the same feature, we are still left with the need to transform the features in 
such a way as to provide commensurability in intensity space. We have not envisioned an 
alternative since the objective is to provide an intensity/angle/angle image for a pilot. The 
fusability issue is also linked with the issue of deciding which sensor contributes how much to 
the final image. The visible IR fusion effort used a binary decision rule, but we anticipate that 
a blending rule will prove advantageous in the present case. Some departure from current 
radar practice may be needed to assess the image quality of the radar signal, and assign the 
transformed image an equivalent intensity for a blending rule. 
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Still another issue of concern is the subject of clutter. Spatial clutter is a potential problem for 
both sensors, while temporal clutter is observed in Radar images. Such clutter complicates the 
processing task, since it represents additional features which must be analyzed. Applying image 
extrapolation to achieve compatibility with the 30 Hz video, may aggravate clutter as a 
distraction to the pilot. The low sample rate which is provided by the radar is effectively 
aliased into higher temporal frequencies by any extrapolation algorithm. 

Current Plans 


Westinghouse is engaged in an analytic and experimental program to investigate these issues. 
The analytic program includes development of basic theoretical models for the sensor 
phenomenology, as well as investigations using simultaneous data from multiple sensors. To 
address these issues requires that a significant data base be available. Westinghouse has an 
instrumented aircraft that provides both radar and E-O sensors with digital data collection. 
Initial efforts will include collecting data from the Westinghouse MODARS weather radar 
together with visible and IR E-0 data. This will be processed in our image processing 
laboratory to evaluate algorithms and assess fundamental problems which must be solved. 
From these results, we plan to formulate a program where the fusion process can be 
implemented as a real time airborne process. 

Conclusions 

The fusion of Radar and E-0 sensor data will provide the ability to select an optimum mix of 
resolution and penetration for each weather condition that will be encountered. To be effective, 
the fundamentals of fusion across different image domains must be established so that a fully 
automated fusion system can be implemented. The spatially coherent pyramid decomposition 
technique appears to offer significant benefits in this fusion effort. There are fundamental 
unanswered questions which must be addressed. In addition, the experimental data base 
required to assess alternative theories has not been obtained. Westinghouse has initiated a 
program that will address the theoretical and experimental issues of Radar E-0 fusion. 
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Figure 1 Visible Sensor Image of Runway and Environs 



Figure 2 Angle/ Angle Radar Image of Runway and Environs 
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Figure 3 Azimuth/Range Radar Image of Runway and Environs 
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Figure 4 Conversion from Radar Range to Elevation 


230 





Figure 5 Radar/E-0 Fusion Using Pyramid Transform 



Figure 6 Reduced E-O Resolution Matches Radar Resolution 
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Figure 7 Generalized Radar E-O Fusion Using Transforms 


232 










IV. IMAGE PROCESSING: COMPUTER VISION 



